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Workshop Charge 


e Develop a written (living) inventory of existing and 
planned/budgeted NASA/ESD Near-Real-Time data sets, 
information products, and associated discovery/distribution 
infrastructure 


e ldentify any significant populations of supporters and/or potential 
benefits from our existing measurements and data sets, that are 
not being fulfilled because NASA/ESD products are being made 
available with too much delay 


Definitions ..... for the purpose of this presentation 


Low latency, or Near real-time (NRT), data are made 
available much faster than routine processing allows. 
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Data latency refers to the time 
between data acquisition and 
the time the data is available to 
the end-user. 

Defining latency? 

O-1 hour = Real-time 

< 3 hours = Near real-time 

3 — 24 hours = Low latency? 

> 24 hours but quicker than routine = expedited? 


Meeting Overview 


75 NASA 


11 Universities 


2.5 day meeting 104 Participants 
Plenary sessions 11 Other US government 


¢ NASA Headquarters 4 Private Sector 
ae oe Non-Government 

e Existing systems for providing NRT data 

e Selected data providers and users 

Breakout Groups 


¢ Completing the Inventory 


e Identifying opportunities and gaps in the provision and delivery 
of NRT data 


ee, 


NASA HQ Perspective: “Low-Latency” Principles in 
Earth Science Division Portfolio Development 


NASA Earth Science Division (ESD) endeavors to provide 
low-latency data and products when it involves small 
additional costs 


NASA ESD is primarily a science organization and science 
and non-real-time research objectives will outweigh 
support for near-real-time users when they are in direct 
conflict. 


NASA Mission 
perspective on 
support for NRT data 
production 


lf NRT considerations should be part of 
the project trade space then 
cost/benefit should be considered 
when the project is in it’s embryotic 
stage 


After design starts don’t expect 
changes in latency design 


Latency can be improved during 
operations, but a non-project funding 
source will need to be identified. 


Inventory 


Develop a written 
“living” inventory of 
existing and 
planned/budgeted 
NASA/ESD Near-Real- 
Time data sets, 
information products, 
associated discovery 
and distribution 
infrastructure. 


http://tinyurl.com/nhmv9ky 


Existing Missions 


Aqua 

Aqua 

Aqua 

Aura 

Aura 
CALIPSO 
GCOM-W1 
GPM 

GRACE 
S-NPP 
S-NPP 
S-NPP 
S-NPP 

Terra 

Terra 

Terra 

Terra 

Terra 
JASON-2, 3 
Landsat 7 
Landsat 8 
SMAP 

ISS Missions 
ISS Missions 
ISS Missions 
ISS Missions 


Geostationary 
GMAO Models 


Instrument 


CERES 

MODIS 

AIRS 

OMI 

MLS 

CALIOP 
AMSR2 
GMI/Multiple 
Multiple 
ATMS 

CERES 

VIIRS 

OMPS 

ASTER 

CERES 

MODIS 
MOPPIT* 
MISR 
Poseidon-3, 3b 
ETM+ 
OLI/TIRS 
L-band Passive 
CATS 

LIS* 

HDEV 


Crew 
Observations 
AHI GOES 
GOES 5 
processing 


TOTAL 


Number of 
products 


20 


Ww 
Suns 


PF PN BN BP PB WwW PB 


ep oO 


173 


Latency Term 


Expedited 
NRT 

NRT 

NRT 

NRT 
Expedited 
NRT 
NRT-Expedited 
Expedited 
NRT 
Expedited 
NRT 

NRT 

Low Latency 
Expedited 
NRT 

NRT 

NRT 
Expedited 
Low Latency 
Low Latency 
Low Latency 
Low Latency 
Real time 
Real time 
Expedited 


Real time 
NRT 


Latency 
Range 


4-5 days 
<3 hrs 
<3 hrs 
<3 hrs 
<3 hrs 
6-30 hrs 
<3 hrs 

1 hr - 16 hrs 
monthly 
<3 hrs 
4-6 days 
<3 hrs 
<3 hrs 
<24 hrs 
4-6 days 
<3 hrs 
<3 hrs 
<3 hrs 
<24 hrs 
4-8 hrs 
4-8 hrs 
4-7 hrs 
4-7 hrs 
<30 min 
<30 min 
24+ hrs 


5-30 min 
6 hrs 


Inventory: http://tinyurl.com/nhmv9ky 


| Mission Instrument Descri SHORT NAME NAME DATA PROVIDER LATENCY Dr if riculture  Smow/lce mont ocean / take c disturbance ¢ Wildlarc fire Smobe Plu 
‘ONGOING MISSIONS 
O_O 
| Dawn CERES Surface and Cloud Properties (SSF) FLASH SSF Aqua CERES/ASOC 45 days x 
i TOA surface fluxes and clouds (SSF) FLASH_SSF_Aqua CERES/ASOC 45 days x x 
| TOA and Surface radiative flux averages (daily aver: FLASH _TISA [Terra+Aqua} CERES/ASOC 5-6 days x x 
{ Synoptic radiative fluxes FLASH_TISA [TerratAqua} CERES/ASOC 5-6 days x x 
i Synoptic cloud and water vapor FLASH TISA +: cE ‘ASDC 5.6 da’ x 
' 
aqua moos Raw radiances MyDOo1 LANCE. MODIS <3 hours x 
J Geolocation myYDo3 LANCE MODIS <3 hours x x 
' Calibrated radiances MYDZ21KM/HKM/OKM LANCE. MODIS <3 hours x x x 
3 Acrosols MYDO4 12 LANCE. MODIS <3 hours x x 
J Total Precipitable Water Vapor MYDOS_t2 LANCE. MODIS <3 hours 
i Clowd Product MYDO6 12 LANCE. MODIS <3 hours x x 
j Temperature and Water Vapor Profiles MYDO7_12 LANCE. MODIS <3 hours x x 
i Land Surface Reflectance mybos LANCE. MODIS <3 hours x x x x 
i Land Surface Reflectance, coarse MYDOSCRS LANCE. MODIS <3 hours + 
} Acrosol optical thickness MyYDOSCMA LANCE MODIS: x x 
' Surface reflectance, climate modeling grid MYDOSCMG LANCE. MODIS 
‘ Surface reflectance, L2G Lite MYDO9GA LANCE. MODIS x x x 
' 
Daily Surface reflectance, L2G Lite MODOSGST LANCE. MODIS 
1 
; Snow Cover MYD10 12 LANCE. MODIS <3 hours x x x 
! Surface Temperature and emissivity MYD11 12 LANCE, MODIS <3 hours x x x 
} Thermal Anomalies / Fire mMyYD14 LANCE. MODIS <3 hours x x x 
} Thermal Anomalies / Fire text for FIRMS MYD14T LANCE-MOOIS- FIRMS <3 hours x x 
' Sea ice extent MyD23 LANCE. MODIS <3 hours x x 
} Clowd mask MYD35_12 LANCE. MODIS <3 hours x x x x x 
, Value added Acrosol Optical Depth MYDAODHD LANCE. MODIS x 
' Daytime geolocation angles MYDMGGAD LANCE-MODIS 
] Daytime pointers MYDPTHKM LANCE. MODIS 
} Daytime thermal bands MYDTSGD LANCE. MODIS x x 
' Corrected reflectance imagery LANCE. MODIS x 
} Rolling BRDE / Albedo / NBAR* ] LANCE. MODIS x 
, Volcano alert MYDVOLC LANCE. MODIS 
' Rolling 8-day surface Reflectance MYDOSAIN LANCE. MODIS x x 
} Sea Surface Temperature J MODISA L2 SST 06 DAAC 2-4 hours x x x 
; Ocean Color Data MODISA_L2_ OC O08 DAAC 2-4 hours x 
‘ Ocean inherent optical properties MODISA L2 SOP O68 DAAC 2-4 hours x 
] NOVI x x 
, LAI* x x 
' Groen Veg Fraction x x 


In flight icing detection NASA SarCORPS <0 min 


Discoverability and Usability 


Inventory is one step to increasing access to NRT datasets but 
Discoverability and Usability are critical. 


¢  Discoverability includes ways in which the community can determine 
which data are available and where they can be obtained. 


e Usability is the ability to easily visualize or integrate the data into 
analysis tools to facilitate data use 


EOSDIS already has services that can help users discover and 
visualize data. These include: 

¢ Common Metadata Repository (CMR) 

¢ Global Imagery Browse Services (GIBS) 

¢ — Earthdata Search Client 

¢ Worldview 


Common Metadata Repository (CMR) 


@ What does it do? 
oO It manages the metadata catalog for NASA’s Earth 
observations + includes some external collection info 
@ What does it contain? 
© Collection info (e.g., “MODIS/Aqua Near Real Time (NRT) 
Aerosol 5-Min L2 Swath 10km”) 
oO Granule info (e.g., “BeginningDateTime: 
2016-09-26T01:20:00.0000002”) 
oO Parameter info 


Oo Etc. 


CMR metadata supports various standards (such as such as ISO 19115-1, ISO 
19115-2, ECHO 10, DIF 9, or DIF 10) and supports OpenSearch compliant queries 
enabling by other search clients. 10 


Global Imagery Browse Services (GIBS) 


e Visualization 
oO The Global Imagery Browse Services (GIBS) provides full-resolution 


imagery via web services for 200+ EOSDIS collections (a fraction of 
the total) 

oO The Worldview client provides an interface for all GIBS products and 
will continue to categorize its layers into NRT application areas 


nasa \X/ORLDVIEW 
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GIBS Integrations 


Earthdata Search 


Image representations of data granules served by GIBS can be viewed on a map in the 
Earthdata Search client (| 


@ Earthdata Login 


& Retrieve Project Data o 


¥ 


“a MODIS/Aqua Land Surface ‘ 
Temperature/Emissivity 5-Min L2 ‘ 
eu — 
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Search and Order 

© The Common Metadata Repository (CMR) is a database for ALL 
collections (over 30k) available through EOSDIS 

© The Earthdata Search client provides a window into all collections in 
CMR 


@ Back to Collections 


ar Re 


& Retrieve Collection Data =} 


@ Report « metadata problem 


MYD04_3K.A2016267 .1900.006.2016267212044.» 
2016-09-23T19:00.002 to 2016-08-23719.05:00Z 


MYDO4 3K.A2016267.1855.006.2016267201005.! 
2016-08-297 18-55-0072 to 2016-06-25T19:00-002 


pay 
MOD#tS/Aqua Near Real Trne (NRT) Aerosol 5-NMin L2 Swath 


13 


Existing Approach https://search.earthdata.nasa.gov/ 


Browse Collections 
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Near Real Time x NRT metadata problem 


MODIS/Aqua Near Real Time 
(NRT) Aerosol 5-Min L2 Swath 
10km 

MYDO4 L2 v6GNRT 
NASA/GSFC/EOS/ESDIS/LANCEMODIS 


Subsetting Services 


Keywords / \ 


om) 
Atmosphere 
Biosphere L 


Cryosphere 


2015-12-06 
ongoing 1549 Granules 


eo | + 


MODIS/Aqua Near Real Time 
(NRT) Land Surface 
Temperature/Emissivity 5-Min 


Land Surface 


Oceans L2 Swath 1km (C005) 
MYD11_L2 vSNRT « 
ele) (em =t- lata) NASA/GSFC/EOS/ESDIS/LANCEMODIS 


2002-05-04 a 


; Oo | + 
Spectral/Engineering ongoing | 2823 Granules 


Terrestrial Hydrosphere 


MODIS/Aqua Near Real Time ¢ 1000 km 
(NRT) Snow Cover 5-Min L2 500 mi 
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https://worldview.earthdata.nasa.gov/ 


Recommendations on Data Discoverability 


Require all NASA programs producing NRT Earth observation data to register 
data in the CMR at the collection level 
Systematic modeling products, such as GMAO products should add imagery 


to GIBS as appropriate 

Develop a set of managed CMR metadata tags to i) distinguish low latency 
products from other products and ii) utilize keywords from NASA’s Global 
Change Master Directory to identify application areas and facilitate searches 
by application area 

Develop a process to evaluate applications products, value added low-latency 
products that use NASA data and non-NASA satellite data. 

¢ Those that met this criteria should be added to CMR at the Collection 


level 


Recommendations on how to engage new missions 


New missions should survey 
communities to determine the 
value of low latency products. 


If NRT products would be useful, 
missions should conduct a latency 
trade study — to help determine 
cost / benefit. 

ESD flight projects should add 
latency to Earth Venture 
solicitations and directed mission 
requirements. 


Recommendation to form a Latency Working Group 


To carry forward recommendations and conduct strategic 
engagement and planning 


° engage with programs and be an 
advocacy group for the NRT data 
producers 

e review the progress towards 
improved data discoverability and 
usability 
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Next Steps 


Latency Working Group 


Register all systematic NRT NASA data in CMR and by default 
into the Earthdata Search Client (and other data portals) 


3. Determine criteria and pathway for deciding how non- 
systematic if other products should be added in to CMR 


To view the NRT Inventory: http://tinyurl.com/nhmv9ky 


Contact? support@earthdata.nasa.gov with NRT workshop in the subject line 


Thank you 


To view the inventory of existing and planned/budgeted NASA/ESD 
Near-Real-Time data sets go to: http://tinyurl.com/nhmv9ky 


